Databricks Community

Confused · 04-04-2022

Hi I would like to use the azure artifact feed as my default index-url when doing a pip install on a Databricks cluster. I understand I can achieve this by updating the pip.conf file with my artifact feed as the index-url. Does anyone know where i...

Confused · 01-11-2022

Hi AllI have a requirement to perform updates on a delta table that is the source for a streaming query.I would like to be able to update the table and have the stream continue to work while also not ending up with duplicates.From my research it se...

Confused · 12-03-2021

Hi AllI am loading some data using auto loader but am having trouble with Schema evolution.A new column has been added to the data I am loading and I am getting the following error:StreamingQueryException: Encountered unknown field(s) during parsing:...

Confused · 11-16-2021

Hi GuysIs there any documentation on where the /databricks-datasets/ mount is actually served from?We are looking at locking down where our workspace can reach out to via the internet and as it currently stands we are unable to reach this.I did look ...

Confused · 12-03-2021

Yeah I get the same error, ran the job twice per the docs as the first should fail then second succeed and identical error.

Confused · 12-03-2021

Hmmmm, I hadn't added it as that doc says it is a default when you don't provide a schema.addNewColumns: The default mode when a schema is not provided to Auto Loader. I will give it a try though, thanks.

Confused · 12-03-2021

Hi JosephkI had read that doc but I don't see where I am having an issue.Per the first example it says I should be doing tthis:spark.readStream.format("cloudFiles") \ .option("cloudFiles.format", "json") \ .option("cloudFiles.schemaLocation", "<pat...

Confused · 11-17-2021

Hi @Jose Gonzalez That's not correct, for us at least, we are using Azure and the mount goes off to AWS S3. It also doesn't give us the information when we run that command, it was the first thing I tried:As you can see, it doesn't give us the real...

Confused · 11-16-2021

Hi HubertThanks, but I guess I didn't explain the issue well, we were looking to find out what was mounted to that location. We've got it now, it goes off to an S3 container in AWS.CheersMat

Databricks Community

User Stats

User Activity

Configuring pip index-url and using artifacts-keyring

Dealing with updates to a delta table being used as a streaming source

Schema evolution issue

Hi Guys Is there any documentation on where the /databricks-datasets/ mount is actually served from?We are looking at locking down where our workspace...

Re: Schema evolution issue

Re: Schema evolution issue

Re: Schema evolution issue

Re: Hi Guys Is there any documentation on where the /databricks-datasets/ mount is actually served from?We are looking at locking down where our workspace...

Re: Hi Guys Is there any documentation on where the /databricks-datasets/ mount is actually served from?We are looking at locking down where our workspace...