- 9654 Views
- 2 replies
- 3 kudos
Resolved! drop specific partition from a Delta Table
i have a delta table partitioned by a Date column , I'm trying to use the alter table drop partition command but get ALTER TABLE DROP PARTITION` is not supported for Delta tables erroris there a way to do it?
- 9654 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Hanan Segal Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 3 kudos
- 6812 Views
- 1 replies
- 2 kudos
Running into delta.exceptions.ConcurrentAppendException even after setting up S3 Multi-Cluster Writes environment via S3 Dynamo DB LogStore
My use-case is to process a dataset worth 100s of partitions in concurrency. The data is partitioned, and they are disjointed. I was facing ConcurrentAppendException due to S3 not supporting the “put-if-absent” consistency guarantee. From Delta Lake ...
- 6812 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, You can refer to https://docs.databricks.com/optimizations/isolation-level.html#conflict-exceptions and recheck if everything is alright. Please let us know if this helps, also please tag @Debayan with your next response which will notify me, Th...
- 2 kudos
- 10518 Views
- 1 replies
- 2 kudos
Resolved! Compiling Flattened Dataframe back to Struct Columns
I have a dataframe with this format of columns:[`first.second.third` , `alpha.bravo.test1` , `alpha.bravo.test2`]I'd like to get an output dataframe of this:[ `first` | `alpha` ] ---------------...
- 10518 Views
- 1 replies
- 2 kudos
- 2 kudos
I have figured out the solution.
- 2 kudos
- 2567 Views
- 3 replies
- 0 kudos
Is there a utility to convert between "/dbfs" and "dbfs:" path strings?
Is there a built-in utility function, e.g., dbutils, that can convert between path strings that start with "dbfs:" and "/dbfs"?Some operations, e.g, copying from one location in DBFS to another using dbutils.fs.cp() expect the path starting with "/db...
- 2567 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Fijoy Vadakkumpadan Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best a...
- 0 kudos
- 11318 Views
- 11 replies
- 7 kudos
Random error: At least one column must be specified for the table?
I have the following code in a notebook. It is randomly giving me the error, "At least one column must be specified for the table." The error occurs (if at all it occurs) only on the first run after attaching to a cluster.Cluster details:Summary5-1...
- 11318 Views
- 11 replies
- 7 kudos
- 7 kudos
Please check if this could help or not:spark.databricks.delta.catalog.update.enabled false
- 7 kudos
- 7641 Views
- 1 replies
- 1 kudos
bucket ownership of s3 bucket in databricks
We had a databricks job that has strange behavior,when we passing 'output_path' to function saveAsTextFile and not output_path variable the data saved to the following path: s3://dev-databricks-hy1-rootbucket/nvirginiaprod/3219117805926709/output_pa...
- 7641 Views
- 1 replies
- 1 kudos
- 1 kudos
I suspect you provided a dbfs path to save the data hence the data saved under your workspace root bucket.For the workspace root bucket, databricks workspace will interact with databricks credential to make sure databricks has access to it and able t...
- 1 kudos
- 1817 Views
- 1 replies
- 0 kudos
Unable to create bloom filter index
I am unable to create bloom filter index on my tableCREATE BLOOMFILTER INDEX ON TABLE my_namespace.foo FOR COLUMNS (id OPTIONS (fpp = 0.1, numItems = 6000000))Gives the errorAnalysisException: Table `spark_catalog`.`my_namespace`.`foo` did not specif...
- 1817 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can refer to https://issues.apache.org/jira/browse/SPARK-27617 for the above error. Please let us know if this helps, also please tag @Debayan with your next response which will notify me, Thank you!
- 0 kudos
- 8495 Views
- 1 replies
- 1 kudos
DLT Spark readstream fails on the source table which is overwritten
I am reading the source table which gets updated every day. It is usually append/merge with updates and is occasionally overwritten for other reasons. df = spark.readStream.schema(schema).format("delta").option("ignoreChanges", True).option('starting...
- 8495 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, Could you please confirm DLT and DBR versions? Also please tag @Debayan with your next response which will notify me, Thank you!
- 1 kudos
- 6076 Views
- 6 replies
- 0 kudos
Why the SFTP ingest doesn't work?
Hi, I did the following code but it seems like the cluster is running for a long period of time and then stops without any results. Attached my following code: (I used 'com.springml.spark.sftp' library and install it as Maven)Also i whitelisted my lo...
- 6076 Views
- 6 replies
- 0 kudos
- 0 kudos
@Debayan Mukherjee Hi, I don't know if you got my reply so i am bouncing my message to you again.Thanks.
- 0 kudos
- 4091 Views
- 2 replies
- 3 kudos
Resolved! Column is accessible after dropping the same column
Hi Today I have seen very Strang behavior of databricks.I have dropped one column from a dataframe and assigned the result to a new dataframe but I am able to use the dropped column in the filter command.In general scenario I should get an error but ...
- 4091 Views
- 2 replies
- 3 kudos
- 3 kudos
@Ajay Pandey , this is a known behavior. Please refer this JIRA for details: https://issues.apache.org/jira/browse/SPARK-30421
- 3 kudos
- 7083 Views
- 15 replies
- 4 kudos
Welcome 2023 Virtual hackathon participants, we're happy to have you! ✋ Please use this space to ask questions, we'll have some folks from Da...
Welcome 2023 Virtual hackathon participants, we're happy to have you! Please use this space to ask questions, we'll have some folks from Databricks and the community join to help out. We're really excited to see what you work on and if you have any ...
- 7083 Views
- 15 replies
- 4 kudos
- 4 kudos
Hi, I also have another question. Do we get any Databricks platform access for the period of hackathon? My company didn't use one and the trial is only 14 days. I'm pretty aware of the capability and since the hackathon are held by Databricks I think...
- 4 kudos
- 1264 Views
- 1 replies
- 1 kudos
Resolved! How is brainstorming going?
Wondering if anyone is willing to share their project ideas here. It would be great to know how things are going and if anyone has a good open-source dataset they are willing to share.
- 1264 Views
- 1 replies
- 1 kudos
- 1 kudos
Good, read their docs to get a lot of info to sharpen this hackathon
- 1 kudos
- 1025 Views
- 1 replies
- 0 kudos
- 1025 Views
- 1 replies
- 0 kudos
- 0 kudos
You just import it. Did you add the library to your cluster?
- 0 kudos
- 1263 Views
- 1 replies
- 0 kudos
Databricks to redash
We have an issued security in redash, if we used databrick as a connector to redash, Can you support us?https://www.databricks.com/blog/2020/06/24/welcoming-redash-to-databricks.html
- 1263 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Probis Useetv Thank you for reaching out to us.Would you please elaborate your use case about the "issued security in redash" ?
- 0 kudos
- 3254 Views
- 3 replies
- 3 kudos
Resolved! Generating an Account console PAT token
I can't seem to find any documentation on generating an account console PAT token, Can anyone link me to it or guide me?
- 3254 Views
- 3 replies
- 3 kudos
- 3 kudos
You can't create a Personal Access Token on account level to use REST APIs. If you want to use SCIM on account level, on the account console settings you'll find the user provisioning tab. In there you can generate de SCIM token. If you want to acces...
- 3 kudos
| User | Count |
|---|---|
| 1612 | |
| 769 | |
| 348 | |
| 286 | |
| 252 |
