- 3000 Views
- 1 replies
- 4 kudos
pyspark delta table schema evolution
I am using the schema evolution in the delta table and the code is written in databricks notebook. df.write .format("delta") .mode("append") .option("mergeSchema", "true") .partitionBy("date") .save(path)But I ...
- 3000 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @z yang​ Please provide the df creation code as well to understand the complete exception and scenario.
- 4 kudos
- 4799 Views
- 1 replies
- 4 kudos
Best practice to delete /dbfs/tmp ?
What is best practice regarding the tmp folder? We have a very large amount of data in that folder and not sure whether to delete, back up etc?
- 4799 Views
- 1 replies
- 4 kudos
- 4 kudos
/dbfs/tmp can contain a lot of files including temporary system files used for intermediary calculations or other sub directories which can contain packages of user defined installations. It is always better to backup the files.
- 4 kudos
- 8012 Views
- 3 replies
- 6 kudos
Unable to write Data frame to Azure Synapse Table
When I am trying to insert records into the azure synapse Table using JDBC Its throwing below error com.microsoft.sqlserver.jdbc.SQLServerException: The statement failed. Column 'COMPANY_ADDRESS_STATE' has a data type that cannot participate ...
- 8012 Views
- 3 replies
- 6 kudos
- 6 kudos
Columns that use any of the following data types cannot be included in a columnstore index:nvarchar(max), varchar(max), and varbinary(max) (Applies to SQL Server 2016 and prior versions, and nonclustered columnstore indexes)so the issue is on the Azu...
- 6 kudos
- 2220 Views
- 1 replies
- 3 kudos
- 2220 Views
- 1 replies
- 3 kudos
- 3 kudos
You can try to refer the example below: https://docs.databricks.com/dev-tools/api/latest/examples.html#upload-a-big-file-into-dbfs
- 3 kudos
- 3925 Views
- 2 replies
- 1 kudos
Resolved! Check Existence of table
Hi Team,How do we check the existence of a table in ADF container using SQL query in Databricks?Thanks in advance.
- 3925 Views
- 2 replies
- 1 kudos
- 2448 Views
- 1 replies
- 3 kudos
Sync prod WS DBs to dev WS DBs
We have a couple sources we'd already set up to stream to prod using a 3p system. Is there a way to sync this directly to our dev workspace to build pipelines? eg. directly connecting to a cluster in prod and pull with a job cluster, dump to S3 and u...
- 2448 Views
- 1 replies
- 3 kudos
- 3 kudos
DBFS can be used in many ways. Please refer below: Allows you to interact with object storage using directory and file semantics instead of cloud-specific API commands.Allows you to mount cloud object storage locations so that you can map storage cre...
- 3 kudos
- 2594 Views
- 1 replies
- 2 kudos
When to use Dataframes API over Spark SQL
Hello Experts,I am new to Databricks. Building data pipelines, I have both batch and streaming data.Should I use Dataframes API to read csv files then convert to parquet format then do the transformation? orwrite to table using CSV then use Spark SQL...
- 2594 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi Rathinam, It would be better to understand the pipeline more in this situation. Writing to table using CSV and then using spark SQL will be faster in few cases than the other one.
- 2 kudos
- 2572 Views
- 2 replies
- 1 kudos
Clarity on usage STREAM while defining DLT tables
Hi, I am currently trying to learn Databricks and going through tutorials and learning materials. I came across this link https://databricks.com/discover/pages/getting-started-with-delta-live-tablesWhile I get most of what is described in page, I fin...
- 2572 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Lokesh Raju​,Just a friendly follow-up. Did Tomasz's response help you to resolved your question? If it did, please mark it as best.
- 1 kudos
- 7072 Views
- 3 replies
- 6 kudos
ENABLE CHANGE DATA FEED FOR EXISTING DELTA-TABLE
I have a delta table already created, now I want to enable the change data feed. I read that I have to set delta.enableChangeDataFeed property to true. But however, this cannot be done using the Scala API. I tried using this but it didn't work. I am ...
- 7072 Views
- 3 replies
- 6 kudos
- 6 kudos
'delta.enableChangeDataFeed' have to be without quotes. spark.sql("ALTER TABLE delta_training.onaudience_dpm SET TBLPROPERTIES (delta.enableChangeDataFeed = true)").show()
- 6 kudos
- 4205 Views
- 2 replies
- 2 kudos
Resolved! Databricks Spark SQL function "PERCENTILE_DISC()" output not accurate.
I am try to get the percentile values on different splits but I got that the result of Databricks PERCENTILE_DISC() function is not accurate . I have run the same query on MS SQL but getting different result set.Here are both result sets for Pyspark ...
- 4205 Views
- 2 replies
- 2 kudos
- 6033 Views
- 5 replies
- 5 kudos
Job fail due to Access Denied
please help me to solve the problem that my data bricks account can not start the Job by triggering manually or scheduling although I can run the script without error.
- 6033 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @trung nguyen​ , Please check if you have the necessary instance profile attached to the Job cluster. You are definitely missing something related to the IAM.
- 5 kudos
- 2956 Views
- 4 replies
- 4 kudos
Invalid shard address
I'm running pyspark through databricks-connect and getting an error saying```ERROR SparkClientManager: Fail to get the SparkClientjava.util.concurrent.ExecutionException: com.databricks.service.SparkServiceConnectionException: Invalid shard address:`...
- 2956 Views
- 4 replies
- 4 kudos
- 4 kudos
hi @Marco Wong​ was this working before and failing now? Are you behind a VPN or firewall? If so can you check by disabling it?enable traces at wireshark and collected dump to check if there is traffic going to workspace?Check if you can get curl wor...
- 4 kudos
- 2417 Views
- 3 replies
- 0 kudos
How do I automate my Databricks script?
How do I automate my Databricks script?
- 2417 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey there @Kayla Simons​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell...
- 0 kudos
- 4834 Views
- 3 replies
- 2 kudos
What are the best practices for schema drift using Delta Live tables, in a scenario where the main source is a no sql database and we have a lot of ch...
What are the best practices for schema drift using Delta Live tables, in a scenario where the main source is a no sql database and we have a lot of changes in the schema?​
- 4834 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey there @Fernando Martin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...
- 2 kudos
- 10683 Views
- 10 replies
- 4 kudos
Databricks job fails while creating table.
Hi Team,The Databricks job fails with the below error while creating EXTERNAL table.com.simba.spark.jdbc41.internal.apache.http.wire - Error running query: MetaException(message:Got exception: org.apache.hadoop.fs.azure.AzureException com.microsoft.a...
- 10683 Views
- 10 replies
- 4 kudos
- 4 kudos
Hey there @Rajesh Kannan R​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
