Plugin: FeedGrab

You’re absolutely right - the problem was that I had a stray </fieldname> in my feed that was causing it to choke.

I’ve now got it working when there are a small number of items in the feed, but my real feed contains around 1000 items and it doesn’t seem to like this. Any idea if this plugin can handle that volume?

I always tend to reduce things to groups of 100, since I find it easier to spot errors or other issues, but I would assume it’s doable unless you get a time out error. My import of 100 records takes around 1min. But it hits a remote feed that can have other timing issues.

I am wondering something before I use the plugin. I have a custom field which I need static data inserted into when the plugin fetches the rss. Example of the field name is “section1”, and I need to insert the word “external_news” into each new recorded created. Can I do this like this?

{exp:feedgrab url="http://example.com/news"
weblog="1"
title="title"
date="dc:date.Taken"
use="link|description|external_news"
fields="extended|body|section1"
}

No. At least not in the number of methods I tried to do so. But you can use categories, which is what I did to handle feed source info.

thanks allgood2! Well categories are not anywhere near how I have things setup, I guess I will have to write a query to update those fields if empty:(

I’m trying to use FeedGrab for the first time, and everything seems to be working pretty well, except that it is stripping out certain characters when it’s importing data. For example, a URL inside a post goes from this:

http://www.somesite.com/blog/atom-feed.php?cat=1234&section=9

to this:

http://www.somesite.com/blog/atom-feed.php?cat=1234section=9

Also… it is stripping out all HTML ascii tag syntax. So any ascii less-than and greater-than signs are missing once they are imported into the weblog.

Any ideas what’s causing this, or how I might work around it?

Hi Andrew, your plugin is really great. Lately I ran into some problems and tried to find the answer in the forums. I found this post in this thread

Notice: Undefined index: date in ~/pi.feedgrab.php on line 221

My problem is similar. Feedgrab reports

Notice: Undefined index: date in /plugins/pi.feedgrab.php on line 569

All feeds are coming from Feedburner. feedgrab grabs them, but there are issues with the date, because when I use the exp:weblog:entires tag as shown below, the order of the items is wrong. When I clear the database and let feddgrab pull the feeds again sometimes the order is correct and sometimes not.

{exp:weblog:entries weblog="news" orderby="date" limit="10" show_future_entries="yes" author_id="2"}

I checked the source of the feeds and replaced some instances of

date="pubDate"

with date=”published” or date=”updated”, since some feeds seem to use this tags for the date. But I had no luck.

So, I´m wondering, if somebody can help me here.

Thanks awa

Thank you for creating FeedGrab! I used it with some additional rss fields like comments (used to hold a phone number), needed to bring over more custom fields from one install to another. That is how I needed to use it this time, I set up an RSS feed for my listings on one site and grabbed what I needed and it work seemlessly!!!

sorry for cross post to “how to” 10 days and no response so..

I’ve got a Feedgrab issue where a twitter feedgrab import is not working as it should when output to weblog:entries

I have the “FeedGrab” plugin pulling in data from twitter.

it looks more or less like this:

{exp:feedgrab url="http://twitter.com/statuses/user_timeline/11111111.rss" 
weblog="21" 
title="title" 
date="pubDate" 
use="link|description" 
fields="twtlink|twtbody" 
unique="twtlink" }

twtlink and twtbody are custom fields within a field group dedicated to twitter feeds.

twtbody is a text input field of 161 characters w text formatting set to none twtlink is a text input field of 256 characters w text formatting set to xhtml

the import appears to succeed pulling in 20 or so items, generating a response in the following format:

Fetching: http://twitter.com/statuses/user_timeline/111111.rss Checking: twitterusername: twitter post data Found new entry: twitterusername: twitter post data

the data is visible and editable in the CP. new posts may be entered directly in the CP

the issue comes when I try to use it in an entry form as follows:

{exp:weblog:entries weblog="tweet_feed" orderby="date" limit="10" dynamic="off" show_future_entries="yes"}

{title}<br>
{twtbody}<br>
{twtlink}
{/exp:weblog:entries}

entries entered directly in the CP render fine. entries pulled in via FeedGrab - while they look fine on import and in the CP - do not render…

any bright ideas as to what is happening and how to solve it?

thanks!

Has anyone been using this plugin to make entries from an RSS feed and relate those entries to existing entries?

I guess you could do this on by writing the exact numbers of a relate entry (if all of the things in the feed relate to that entry). But has anyone done this dynamically? I’d like to be able to relate entries I pull from a RSS feed individually to different entries on my site, but I’m not sure this has the capacity to do this.

Todd

The way I’ve done this so far:

Created one weblog for channel level information. Wrote a plugin to update that based on some parameters.

Created a second weblog for item level information. I use Feedgrab here to pull in items and create the database entries.

I modified Feedgrab to accept another parameter, the entry_id for the Feed level weblog item that it is the child of. I also added the ability for it to make a relationship between the item level weblog entries and the feed level weblog entries.

In the page that updates (which I’m going to set as a cron job), I loop through the entities of the Feed weblog, and dynamically pass the parameters (feed url, feed weblog entry id) to the Feedgrab plugin. So basically, all you have to enter in the Feed level info is a title and the url of the feed, and it will pull in the channel level and child item level information.

An incredibly detailed response to something I felt was possibly garbled and unintelligible. Bafadam - if you’re up for your sharing your mods to the plugin, I’d love to see them.

In between, I attempted to see if an embed would make this work, but no such luck.

It seems like this would require some soft of rewrite of the actual plugin to make what it do what I want it to.

Unless anybody else has an answer of how one could automatically relate pulled in feed entries dynamically to different weblog entries with the plugin as is.

Todd

The way I’ve done this so far: Created one weblog for channel level information. Wrote a plugin to update that based on some parameters. Created a second weblog for item level information. I use Feedgrab here to pull in items and create the database entries. I modified Feedgrab to accept another parameter, the entry_id for the Feed level weblog item that it is the child of. I also added the ability for it to make a relationship between the item level weblog entries and the feed level weblog entries. In the page that updates (which I’m going to set as a cron job), I loop through the entities of the Feed weblog, and dynamically pass the parameters (feed url, feed weblog entry id) to the Feedgrab plugin. So basically, all you have to enter in the Feed level info is a title and the url of the feed, and it will pull in the channel level and child item level information.

An incredibly detailed response to something I felt was possibly garbled and unintelligible. Bafadam - if you’re up for your sharing your mods to the plugin, I’d love to see them.

They’re neither fancy nor flexible. The first thing I did was write a completely separate function that swaps out custom field names for actual field names.

function customFieldSwap($arrCustomFields, $weblogId)
    {
        global $DB;
        
        // Get custom fields from database
        $customfieldsArray = $DB->query("SELECT exp_weblog_fields.field_id, 
                                exp_weblog_fields.field_name, 
                                exp_weblog_fields.field_label, exp_weblog_fields.field_fmt
                                 FROM exp_weblogs, exp_weblog_fields
                                 WHERE exp_weblogs.field_group = exp_weblog_fields.group_id
                                AND exp_weblogs.weblog_id = '".$weblogId."'");
        
        // Map weblog custom fields to id's
        $data = Array();
    
        foreach ( $customfieldsArray->result as $row ) {
        
            if (array_key_exists($row['field_name'], $arrCustomFields))
            {
                $fieldid = 'field_id_' . $row['field_id'];
                $fieldname = $row['field_name'];
                $data[$fieldid] = $arrCustomFields[$fieldname];            
            }
        } 
        return $data;
    }

It accepts an array of (customFieldName => customFieldData) and returns an array with (actualFieldName => actualFieldData). It’s not very flexible, though. Since title, pubdate, etc aren’t custom field names, they won’t get swapped, so if you’re using those fields in there, add them after you do your swapping. I suppose it could be updated to deal with system fields, but I’m honestly too lazy for that business, so… moving on.

Andrew is a better programmer than I am. So my code is infinitely less flexible than his, so bear that in mind as you continue reading on.

I added a new class variable called $parent_weblog_id. I created a new fetch param line for, you guess it, $parent_weblog_id. When you call your function, it’s now looking for the entry id of the related (I keep calling it parent) weblog entry.

Since I’m looping through the Feed level entries, the idea is that I’ve got access to this field pretty handily and can just pass it in real easy-like. Like I said, inflexible.

After this line:

$sql = $DB->insert_string('exp_weblog_data', $data);
                if ( $this->runsql ) {
                    $DB->query($sql);
                } else {
                    if ( $this->debug ) { print $sql . "<br>\n"; }
                }

I added in my relationship creation code:

//modification time
                //Let's add our relationship code here.
                //This is a two step process:  1. create the relationship and 2. update the feed item.
                $sql = '';
                
                $relData = array(
                    'rel_parent_id' => $entry_id,
                    'rel_child_id' => $this->parent_weblog_id,
                    'rel_type' => 'blog'    
                );
                
                $sql = $DB->insert_string('exp_relationships', $relData);
                $relID = '';
                
                if ( $this->runsql ) {
                    $DB->query($sql);
                    $relID = $DB->insert_id;
                } else {
                    if ( $this->debug ) { print $sql . "<br>\n"; }
                }
                
                //Apparently, our relationship thing was successful, so let's update the weblog data now.
                if ($relID != '') {
                    $relData = array(
                        'feed_id' => $relID
                    );
                    
                    $relDataSwap = $this->customFieldSwap($relData, 3);
                    
                    $sql = $DB->update_string('exp_weblog_data', $relDataSwap, 'entry_id = ' . $entry_id);
                
                    if ( $this->runsql ) {
                    $relID = $DB->query($sql);
                    } else {
                        if ( $this->debug ) { print $sql . "<br>\n"; }
                    }
                    
                
                }

Andrew, this whole thing is your creation. If me posting any mod code to it bothers you, just let me know and I’ll yank it. You did some awesome work with this whole thing and I don’t want to piss you off. :)

I have used XMLGrab to import content from an old CMS to EE, and the entries are showing in EE fine - everything’s there that should be, and I have been able to go in and edit an ‘old’ entry to make sure all the information we now want is there.

However, then on creating a template to attempt to display entries, I am getting the following errors:

Warning: Template::require_once(./admin/modules/weblog/mod.weblog.php) [function.Template-require-once]: failed to open stream: No such file or directory in /home/[...]/admin/core/core.template.php on line 991

Fatal error: Template::require_once() [function.require]: Failed opening required './admin/modules/weblog/mod.weblog.php' (include_path='.:/usr/lib/php:/usr/local/lib/php') in /home/[...]/admin/core/core.template.php on line 991

It doesn’t matter which weblog I am trying to show, and even if I have {title} as the only tag between the weblog tags - this is what I get.

I’ve read through this thread seeing if anyone else has had similar issues, but it appears not - EEK!