- 2554 Views
- 4 replies
- 0 kudos
write to Delta
spark.conf.set("spark.databricks.delta.properties.defaults.columnMapping.mode","name")products_output_path = DA.paths.working_dir + "/delta/products"products_df.write.format("delta").save(products_output_path) verify_files = dbutils.fs.ls(products_ou...
- 2554 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @SSV_dataeng ,Please check with this (you would have to indent it correctly for python)productsOutputPath = DA.workingDir + "/delta/products"(productsDF.write.format("delta").mode("overwrite").save(productsOutputPath))verify_files = dbutils.fs.ls(...
- 0 kudos
- 6657 Views
- 3 replies
- 1 kudos
Can I change Service Principal's OAuth token's expiration date?
Hi,since I have to read from a Databricks table from an external API I created a Service Principal that would start a cluster and perform the operation, to authenticate the request on behalf of the Service Principal I generate the OAuth token followi...
- 6657 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @marchino Please check if this is of your interest https://kb.databricks.com/en_US/security/set-an-unlimited-lifetime-for-service-principal-access-token
- 1 kudos
- 1403 Views
- 1 replies
- 0 kudos
Reserved VM/DBU's
As Some VM's /DBU's reservations are purchased, however, it was underutilized. How can we optimize the underutilization? Do we have any guidelines or best practices?
- 1403 Views
- 1 replies
- 0 kudos
- 0 kudos
We have 5 reserved instances of Azure VMs to run the data bricks cluster jobs. And this is not being utilized efficiently (as per the usage metrics – 1 of the reservation is 10-15% utilized and the other one is 30-40% utilized). Could you please help...
- 0 kudos
- 4133 Views
- 2 replies
- 1 kudos
Data lineage on views
I do not know if this is intended behavior of data lineage but for me it is weird.When I create a view based on two tables the data lineage upstream looks correct. But when I replace the view to only use one of the tables, then data lineage upstream ...
- 4133 Views
- 2 replies
- 1 kudos
- 1 kudos
After some thoughts, i have come to this conclusion:Data lineage on views is working as one should expect. I strongly recommend that this feature is redesigned so it shows the result of the lastest view.
- 1 kudos
- 4728 Views
- 3 replies
- 0 kudos
Unable to Start Clusters on GCP - Clusters Stuck in "CREATING" State
I set up my Databricks Account on GCP via GCP Marketplace and then created my first workspace via the Accounts Console (default Databricks VPC). Everything seemed to be ok until I attempted to create my first cluster. The cluster hangs indefinitely i...
- 4728 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @etlundquist Thank you for posting your question in our community! We are happy to assist you. To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 1389 Views
- 0 replies
- 0 kudos
Not able to login or change password
Hi,I am unable to login to databricks community edition. I have tried changing my password. However, no email is sent on my email id which is : kum.nit7287@gmail.com. Can anyone help?
- 1389 Views
- 0 replies
- 0 kudos
- 7402 Views
- 3 replies
- 0 kudos
Iterative read and writes cause java.lang.OutOfMemoryError: GC overhead limit exceeded
I have an iterative algorithm which read and writes a dataframe iteration trough a list with new partitions, like this: for p in partitions_list:df = spark.read.parquet("adls_storage/p")df.write.format("delta").mode("overwrite").option("partitionOver...
- 7402 Views
- 3 replies
- 0 kudos
- 0 kudos
@daniel_sahalI've attached the wrong snip/ Actually it is FULL GC Ergonomics, which was bothering me. Now I am attaching the correct snip. But as you said I scaled a bit. The thing I forgot to mention is that the table is wide - more than 300 column...
- 0 kudos
- 4165 Views
- 1 replies
- 3 kudos
Resolved! Using DeltaTable.merge() and generating surrogate keys on insert?
I'm using merge to upsert data into a table:DeltaTable.forName(DESTINATION_TABLE).as("target").merge(merge_df.as("source") ,"source.topic = target.topic and source.key = target.key").whenMatched().updateAll().whenNotMatched().insertAll().execute()Id ...
- 4165 Views
- 1 replies
- 3 kudos
- 3 kudos
@Dekova 1) uuid() is non-deterministic meaning that it will give you different result each time you run this function2) Per the documentation "For Databricks Runtime 9.1 and above, MERGE operations support generated columns when you set spark.databri...
- 3 kudos
- 3570 Views
- 3 replies
- 2 kudos
Databricks SQL - Conditional Catalog query
Hi is there a way we can do%sqlselect * from {{ catalog }}.schema.tableWhere `{{ catalog }}` is a template variable extracted/evaluated from either an environment variable, a databricks secret, or somewhere else? (note: not a widget)
- 3570 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @102842 You can use query parameters to perform this - https://docs.databricks.com/sql/user/queries/query-parameters.htmlYou can define the catalog name as a query parameter. You should declare the catalog name parameter as a drop down list, becau...
- 2 kudos
- 1622 Views
- 1 replies
- 0 kudos
Activation Email is not coming up in the email
Activation Email is not showing up in the email. I am not able to start my trial. @Sujitha or @Kaniz - Please help!
- 1622 Views
- 1 replies
- 0 kudos
- 0 kudos
@Sujitha or @Kaniz - Need your help regarding the validate email not reaching inbox / spam folder to activate trial.
- 0 kudos
- 882 Views
- 0 replies
- 0 kudos
Validate email is not showing up in inbox / spam
No email is coming up to validate and start the trial, need help.
- 882 Views
- 0 replies
- 0 kudos
- 5408 Views
- 4 replies
- 0 kudos
Import dbfs file into workspace using Python SDK
Hello,I am looking to replicate the functionality provided by the databricks_cli Python package using the Python SDK. Previously, using the databricks_cli WorkspaceApi object, I could use the import_workspace or import_workspace_dir methods to move a...
- 5408 Views
- 4 replies
- 0 kudos
- 0 kudos
Even, I am looking for a way to bring files present in S3 to Workspace programmatically.
- 0 kudos
- 1424 Views
- 0 replies
- 0 kudos
Big time differences in reading tables
When I read managed table in #databricks# i can see big differences in time spent. Small test table with just 2 records is once loaded in 3 seconds and another time in 30 seconds. Reading table_change for this tinny table took 15 minutes. Don't know ...
- 1424 Views
- 0 replies
- 0 kudos
- 4043 Views
- 1 replies
- 1 kudos
ENDPOINT_NOT_FOUND error coming for /2.0/clusters/list-zones API on Databricks running over GCP
Hi,I am trying to build the ETL data pipelines on databricks workspace that is running over GCP.For automated cluster creation, when I´m trying to access list availability zones REST API of cluster creation, then it is failing with end point not foun...
- 4043 Views
- 1 replies
- 1 kudos
- 4563 Views
- 1 replies
- 1 kudos
To enroll for featured member interview
What is the procedure to enroll ourselves into feature member interview.
- 4563 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @KVNARK We appreciate your interest in becoming a part of our featured member recognition system. Regrettably, at the moment, enrollment is not possible through an application process. Instead, we identify the top contributors based on their ac...
- 1 kudos
