Plugin: CSVGrab

In addition to seconding Sean’s request for a status field, another similarly useful option would be to specify a field for the url_title.

Right now, it’s auto-generated from the title. I ran into a case today where it would have been great to first check for a url_title in the CSV, and then as a fallback use the auto-generated conversion from the title.

Thanks again for all the work Andrew (and Dan for the tweaks).

~cc

I’ve got this working but it only imports one record, then quits. I have a CSV document with about 60 rows of data in it. What should I be looking for in terms of troubleshooting this CSV document?

Perhaps I should phrase my question in a different way. CSVGrab successfully grabs one row in my CSV file, then shuts down. It will not grab a second record. If I reload, it (re)grabs the first row, then shuts down. If I delete several rows and try again, it grabs the fist row and then shuts down.

What would cause this?

Hi timj - questions and suggestions:

Set trace=”true” in your template for more debugging help.

Could you post your CSVgrab template? Could you post some sample data and sample output? What happens if you put a different first row in the data file?

Double-check the custom field names to make sure there are no errors.

Did you upload the csv file from a Mac? If so, did you make sure to do it in text mode so the line endings are OK?

Did you upload the csv file from a Mac? If so, did you make sure to do it in text mode so the line endings are OK?

That was it. Thanks!

This plugin has proven to be helpful time and time again - thanks!

One question I have is whether there’s any way possible to import data from a custom field based on a weblog relationship. Might be a long shot, but if there’s a way or if someone has a suggestion I’d be happy to give it a shot.

Thanks, Manoj

{exp:csvgrab url="http://www.domain.org/acup.txt"
    delimiter="TAB"
    encloser=""
    site_id="1"
    weblog="14"
    category="151"
    author="3"
    title="3"
    use="1|2"
    fields="dir_name|dir_phone1"
    trace="true"
}

I’m stumped!

I’ve tried .CSV and TAB (.txt) as in this example. Working from a Mac (yes transmit is forcing text for the mode) My output looks like this.

TRACE: Array ( [title] => Array ( [is_custom] => 0 [field] => 3 ) [date] => Array ( [is_custom] => 0 [field] => ) [dir_name] => Array ( [is_custom] => 1 [field] => 1 [id] => 18 [format] => none ) [dir_phone1] => Array ( [is_custom] => 1 [field] => 2 [id] => 19 [format] => none ) [dir_phone2] => Array ( [id] => 20 [format] => none ) [dir_url] => Array ( [id] => 21 [format] => none ) [dir_email1] => Array ( [id] => 22 [format] => none ) [dir_email2] => Array ( [id] => 23 [format] => none ) [dir_summery] => Array ( [id] => 24 [format] => none ) [dir_services_1] => Array ( [id] => 25 [format] => xhtml ) [dir_image] => Array ( [id] => 26 [format] => none ) [dir_services_2] => Array ( [id] => 27 [format] => xhtml ) [dir_services_3] => Array ( [id] => 28 [format] => xhtml ) [dir_tags] => Array ( [id] => 29 [format] => none ) )

The file is attached if that helps (I’ll remove it once I solve this puzzle).

I’ve checked file permissions and even tried a server path vs. url like in this example.

Anyone with a fresh set of eyes see what I’m overlooking here?

I’m really excited about the possibilities of this but I’ve got to get it working first.

Thanks.

Hi Kurt, if you don’t see a line that looks something like this:

TRACE: Resource id #49

in the TRACE output, then it is not even opening the file. What was the absolute file path you gave? It should be the actual filepath, such as you would use if you ssh’d into the server and wanted to access the file. If you can ssh in, make sure you can cat the file with that path.

Can you turn on PHP error reporting to see what’s going wrong with the file open or fetch? That info may also be in the server Apache error logs.

OK… we can see the file

TRACE: Resource id #51

That turned out to be a permissions issue (up loaded that file too many times and lost track).

{exp:csvgrab url="/var/www/vhosts/domain.org/httpdocs/acup.txt"

But it doesn’t seem to matter what path I use to get there (the URL or the sever path), its a no-go.

Ahh… from the sever log:

[Sat Oct 25 23:17:24 2008] [error] [client 00.000.000.000] PHP Warning: fgetcsv() [function.fgetcsv]: enclosure must be a character in /var/www/vhosts/domain.org/httpdocs/system/plugins/pi.csvgrab.php on line 202

At last a clue. It doesn’t appear to like my enclosure’s character…

AND THE ANSWER IS:

encloser=”SPACE”

Yeeee Haaaa… .

Thanks for the help Dan!

Yeeee Haaaa… .

Great! You’re welcome.

The encloser used to be optional, I think, in some previous version of PHP, but has now become required. However, the actual data is not required to have an encloser. If the encloser is present, then it will be used to quote the data so that you can have delimiter characters inside. So if you have

encloser="SPACE"
delimiter="TAB"

then you are saying you may have data like this:

123<tab>456<tab><space>abc<tab>def<space><tab>xyz

That would break into fields like this. Notice that the third field has an embedded tab as part of the data:

"123"    "456"    "abc<tab>def"    "xyz"

Since you are using tab-delimited data, it may be safer to use

encloser="QUOTE"

or some character you’re not expecting at all, just in case one of your fields starts with a space. If it did, that field eat up the rest of the characters, including the following tabs, until there awas another space.

{exp:csvgrab url="http://mine.com/sbs.csv"
    delimiter=","
    encloser="QUOTE"
    skip="1”
    author="1"
    site_id="4" 
    weblog="9"
    category_group="9"
    title="3"
    category_field="1"
    use="1|3|4|5|6|7|8"
    unique="busdir_cat, busdir_address"
fields="busdir_cat|busdir_comp_name|busdir_address|busdir_city|busdir_state|busdir_zip|busdir_phone"
}

Just wanted to make sure that

unique="busdir_cat, busdir_address"

Won’t import any items with the same busdir_cat AND busdir_address right? It’s not an OR comparison..?

If busdir_cat and busdir_address both matched another entry then the item wont be imported, right?

Kevin

Won’t import any items with the same busdir_cat AND busdir_address right? It’s not an OR comparison..? If busdir_cat and busdir_address both matched another entry then the item wont be imported, right?

Yes, they both have to match, so it’s AND. In CSVgrab, an SQL query is generated with equality checks grouped together with AND’s.

Thanks Dan!

edit: i have fixed my own problem. please move along. nothing to see here. :red: