- 2347 Views
- 1 replies
- 0 kudos
Databricks Certified Associate Developer for Apache Spark 3.0 - Python Cancellation
I have scheduled an examination on 1st June 2023 and due to personal reason, I have cancelled the examination on 26th May 2023 (more than 72 hours) but I am yet to receive the refund amount. In the auto generated mail it is mentioned that the refund ...
- 2347 Views
- 1 replies
- 0 kudos
- 0 kudos
adding @Suteja Kanuri and @Vidula Khanna for visibility
- 0 kudos
- 3254 Views
- 1 replies
- 0 kudos
Spark Dataframe write to Delta format doesn't create a _delta_log
Hello everyone, I have an intermittent issue when trying to create a Delta table for the first time in Databricks: all the data gets converted into parquet at the specified location but the _delta_log is not created or, if created, it's left empty, t...
- 3254 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you list (display) the folder location "deltaLocation"? what files do you see here? have you try to use a new location for testing? do you get the same behavior?
- 0 kudos
- 3732 Views
- 1 replies
- 3 kudos
Can't access databricks
I was updated some scripts when I all of the sudden got a few "internal server errors". I refreshed the webpage a couple of times and now I am unable to login to databricks.When I try to sign in it thinks for a few seconds and then I am rerouted back...
- 3732 Views
- 1 replies
- 3 kudos
- 3 kudos
what date and time this issue happened? are you still unable to access to your workspace?
- 3 kudos
- 12113 Views
- 1 replies
- 1 kudos
Resolved! Pydoc / Documentation Module
Does anyone have a recommendation for something along the lines of Pydoc that can be used to aggregate docstrings and the like into documentation pages?I tried Pydoc and it failed because of the magic commands in my repo.
- 12113 Views
- 1 replies
- 1 kudos
- 1 kudos
Sphinx could be an option here. It parses and render Databricks notebooks in the documentation. You might want to look into it to see if it fits your needs. However, it may not handle magic commands very well, and it assumes your notebooks are export...
- 1 kudos
- 3611 Views
- 2 replies
- 3 kudos
Terraform Global Init Script base64encoding
I am working on converting manual global init scripts into a terraform IaC process for multiple environments. Within terraform, we are using the resource "databricks_global_init_script" and set the content_base64 with the following:base64encoded(<<-...
- 3611 Views
- 2 replies
- 3 kudos
- 3 kudos
I am looking into it @Kristian Foster Are you able to get it working?
- 3 kudos
- 11371 Views
- 2 replies
- 3 kudos
Resolved! drop specific partition from a Delta Table
i have a delta table partitioned by a Date column , I'm trying to use the alter table drop partition command but get ALTER TABLE DROP PARTITION` is not supported for Delta tables erroris there a way to do it?
- 11371 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Hanan Segal Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 3 kudos
- 12057 Views
- 1 replies
- 2 kudos
Running into delta.exceptions.ConcurrentAppendException even after setting up S3 Multi-Cluster Writes environment via S3 Dynamo DB LogStore
My use-case is to process a dataset worth 100s of partitions in concurrency. The data is partitioned, and they are disjointed. I was facing ConcurrentAppendException due to S3 not supporting the “put-if-absent” consistency guarantee. From Delta Lake ...
- 12057 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, You can refer to https://docs.databricks.com/optimizations/isolation-level.html#conflict-exceptions and recheck if everything is alright. Please let us know if this helps, also please tag @Debayan with your next response which will notify me, Th...
- 2 kudos
- 11923 Views
- 1 replies
- 2 kudos
Resolved! Compiling Flattened Dataframe back to Struct Columns
I have a dataframe with this format of columns:[`first.second.third` , `alpha.bravo.test1` , `alpha.bravo.test2`]I'd like to get an output dataframe of this:[ `first` | `alpha` ] ---------------...
- 11923 Views
- 1 replies
- 2 kudos
- 2 kudos
I have figured out the solution.
- 2 kudos
- 3842 Views
- 3 replies
- 0 kudos
Is there a utility to convert between "/dbfs" and "dbfs:" path strings?
Is there a built-in utility function, e.g., dbutils, that can convert between path strings that start with "dbfs:" and "/dbfs"?Some operations, e.g, copying from one location in DBFS to another using dbutils.fs.cp() expect the path starting with "/db...
- 3842 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Fijoy Vadakkumpadan Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best a...
- 0 kudos
- 16888 Views
- 11 replies
- 7 kudos
Random error: At least one column must be specified for the table?
I have the following code in a notebook. It is randomly giving me the error, "At least one column must be specified for the table." The error occurs (if at all it occurs) only on the first run after attaching to a cluster.Cluster details:Summary5-1...
- 16888 Views
- 11 replies
- 7 kudos
- 7 kudos
Please check if this could help or not:spark.databricks.delta.catalog.update.enabled false
- 7 kudos
- 8907 Views
- 1 replies
- 1 kudos
bucket ownership of s3 bucket in databricks
We had a databricks job that has strange behavior,when we passing 'output_path' to function saveAsTextFile and not output_path variable the data saved to the following path: s3://dev-databricks-hy1-rootbucket/nvirginiaprod/3219117805926709/output_pa...
- 8907 Views
- 1 replies
- 1 kudos
- 1 kudos
I suspect you provided a dbfs path to save the data hence the data saved under your workspace root bucket.For the workspace root bucket, databricks workspace will interact with databricks credential to make sure databricks has access to it and able t...
- 1 kudos
- 2936 Views
- 1 replies
- 0 kudos
Unable to create bloom filter index
I am unable to create bloom filter index on my tableCREATE BLOOMFILTER INDEX ON TABLE my_namespace.foo FOR COLUMNS (id OPTIONS (fpp = 0.1, numItems = 6000000))Gives the errorAnalysisException: Table `spark_catalog`.`my_namespace`.`foo` did not specif...
- 2936 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can refer to https://issues.apache.org/jira/browse/SPARK-27617 for the above error. Please let us know if this helps, also please tag @Debayan with your next response which will notify me, Thank you!
- 0 kudos
- 9551 Views
- 1 replies
- 1 kudos
DLT Spark readstream fails on the source table which is overwritten
I am reading the source table which gets updated every day. It is usually append/merge with updates and is occasionally overwritten for other reasons. df = spark.readStream.schema(schema).format("delta").option("ignoreChanges", True).option('starting...
- 9551 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, Could you please confirm DLT and DBR versions? Also please tag @Debayan with your next response which will notify me, Thank you!
- 1 kudos
- 9685 Views
- 6 replies
- 0 kudos
Why the SFTP ingest doesn't work?
Hi, I did the following code but it seems like the cluster is running for a long period of time and then stops without any results. Attached my following code: (I used 'com.springml.spark.sftp' library and install it as Maven)Also i whitelisted my lo...
- 9685 Views
- 6 replies
- 0 kudos
- 0 kudos
@Debayan Mukherjee Hi, I don't know if you got my reply so i am bouncing my message to you again.Thanks.
- 0 kudos
- 5887 Views
- 2 replies
- 3 kudos
Resolved! Column is accessible after dropping the same column
Hi Today I have seen very Strang behavior of databricks.I have dropped one column from a dataframe and assigned the result to a new dataframe but I am able to use the dropped column in the filter command.In general scenario I should get an error but ...
- 5887 Views
- 2 replies
- 3 kudos
- 3 kudos
@Ajay Pandey , this is a known behavior. Please refer this JIRA for details: https://issues.apache.org/jira/browse/SPARK-30421
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
