Plugin: CSVGrab

Hitting the sack for now. No errors in the plugin it seems, and this load crapped out at around 11,000 entries. This is the pattern: it gets to 10k-ish entries imported then tanks. Then each subsequent attempt after that only gets to around that 1500 number before dying. I’m wondering if it’s a mysql issue, or even perhaps something more specific to EE.

…and back. After changing further things, still getting the exact same result. Thinking there’s a memory leak somewhere, either in the plugin or in EE itself, that manifests after hitting that 10,000 entry marker…

I had the same issue that it would drop out at about the 10,000 record mark, the only thing I did was chop my csv file into 10,000 record chunks. I’m assuming it’s a timeout problem but as I was in a rush I didn’t have time to play about with Andrew’s code. I did successfully import 560,000 records over the course of weekend though while managing to keep the kids amused at the same time!

Another alternative is to store the data external to the EE tables and then use PHPMyAdmin with the LOAD DATA command. This is a lot faster although I had to reconfigure PHP to load up 10MB CSV files. I loaded up 720,000 records in about 20 mins.

Hi collateit, I did the same thing, splitting the csv down into 5,000 row files. Stil died, but got down to 1400 records at a time. Sheesh. I am/was on Media Temple’s dedicated virtual server package, but their support is slow and offered little in the way of new knowledge. I couldn’t do any db loads from the command line either, as apparently we’re not given those permissions. Ugh. I basically elected to change providers and move to a fully dedicated box now

Update: This is a PHP configuration issue I have since addressed. Feel free to ignore this post. The adventure continues… I have changed hosting providers and am now running on PHP 5 instead of 4. Now I get the following error when I try to run the script:

Fatal error: Call to undefined function mb_convert_encoding() in /path/to/site/plugins/pi.csvgrab.php on line 213

As a new EE user, I’m still confused by the conventions for posting plug-ins in threads.

Some authors update the first post to include a link when a newer version becomes available. Others post the new versions throughout.

This thread, which has many pages, has the newer versions sprinkled throughout.

If there isn’t a convention, could I vote for updating the first post with the current version?

I would also like to vote for the first post update.

I installed the first post plugin (it didn’t work). I then had to read through these pages and then found the newer version of the plugin.

Once I installed the correct version everything worked great! I was only importing 150 articles which was more than I wanted to do manually. I’ll defiantly be using this plugin often. Thank you!

First post would have saved me 1/2 hour.

This worked well for me for about 100,000 entries now, but it’s decided to start grinding again, to the tune of about one entry every two seconds. If I take off the unique setting it zips through pretty well – but instead of inserting duplicates it actually just inserts 1 of the dupe entries. For example, I have several branches of Joe’s Library, but it checks all occurrences and only inserts one of them. I tried a code snippet in an earlier post, but it just borks the script entirely… anyone out there with similar issues?

I only went as far as 50,000 total entries but the plugin still worked for me. When you try to insert too many records at a time, the duplicate entries occur, so if you can limit it to about 3,000 per run, everything should process smoothly.

OK, wow. First, I had already been trying to import a series of smaller files that I had split apart but to the same effect. However, I went into the database itself and modified the datatype of the field I needed to specify as unique (which was a uid in the csv) to a ‘char’ with a length limit of 10. I then created an index on that column. Bang. I’ve imported 30,000 entries now in the time between your response and this one!

Firstly, apologies for neglecting this thread recently. It seemed to have slipped off my “Notify me by email when someone posts” list…

I now (finally) have a new version of CSVGrab that is multiple sites compatible - sorry for the delay in posting. An additional parameter, site_id=”“, has been added to select which site to import the data into.

Also, from now on, rather than posting the plugin in the forum, it now has its own page where the most recent version can be downloaded.

Please see the CSVGrab page on my site for the latest version.

Finally, I’d like to gather some opinions: the usage of the plugin has always been cumbersome and has become increasingly so as features have been added. What would the interest be in a CSVGrab module where the configuration is done using a wizard-style, step-by-step process?

IMHO a wizard style step-by-step process would be user friendly for users who don’t often use the ‘grabber’; however, for the more technical user a step-by-step process may take more time.

Thanks for the update Andrew.

An interface would be nice for this. Perhaps one that has the file upload right in it. This might help offload this work to non-tech users who want to import data. Currently that would be a little cumbersome.

More so though, it would be great to have an expiration date added. Would be nice to have hours and minutes, not just dates as entry and expiration dates.

Thanks for the plugin though, I use it about once every month!

I put the newest version (0.2.1) on my site and now I’m getting this error: Warning: fgetcsv() [function.fgetcsv]: enclosure must be a character in /home/httpd/vhosts/…/pi.csvgrab.php on line 202 I was using 0.1 and it worked great, the only reason I changed it was because I wanted to use the “unique=” and it didn’t seem to work, so I thought maybe that feature wasn’t in the older version. Here is the template I’m calling it with: {exp:csvgrab url=”http://…csv” weblog=”3” title=”1” skip=”1” delimiter=”,” encloser=”” category_field=”4” category_group=”3” use=”2|3” fields=”food-details|food-price” unique=”title,food-details,food-price” }

Like I said, this template works on the same csv with the old version. I didn’t change anything but the csvgrab version and now it’s not working. Any Ideas??