- 6452 Views
- 4 replies
- 4 kudos
Resolved! Schema Parsing issue when datatype of source field is mapped incorrect
I have complex json file which has massive struct column. We regularly have issues when we try to parse this json file by forming our case class to extract the fields from schema. With this approach the issue we are facing is that if one data type of...
- 6452 Views
- 4 replies
- 4 kudos
- 4 kudos
Hey there, @Matt M​ - If @Hubert Dudek​'s response solved the issue, would you be happy to mark his answer as best? It helps other members find the solution more quickly.
- 4 kudos
- 8064 Views
- 9 replies
- 3 kudos
Resolved! Tring to create incremental pipeline but fails when I try to use outputMode "update"
def upsertToDelta(microBatchOutputDF, batchId): microBatchOutputDF.createOrReplaceTempView("updates") microBatchOutputDF._jdf.sparkSession().sql(""" MERGE INTO old o USING updates u ON u.id = o.id WHEN MATCHED THEN UPDATE SE...
- 8064 Views
- 9 replies
- 3 kudos
- 3 kudos
Delta table/file version is too old. Please try to upgrade it as described here https://docs.microsoft.com/en-us/azure/databricks/delta/versioning​
- 3 kudos
- 3187 Views
- 1 replies
- 5 kudos
Resolved! We have hundreds of ETL process (Informatica) with a lot of logic pulling various data from applications into a relational db (Target DB). Can we use Delta Lake as the Target DB?
Hi DB Support,Can we use DB's Delta Lake as our Target DB? Here's our situation...We have hundreds of ETL jobs pulling from these Sources. (SAP, Siebel/Oracle, Cognos, Postgres) .Our ETL Process has all of the logic and our Target DB is an MPP syst...
- 3187 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi yes you can the best is to create sql endpoint in premium workspace and just write to delta lake as to sql. This is community forum not support. You can contact databricks via https://databricks.com/company/contact or via AWS, Azure if you have su...
- 5 kudos
- 2997 Views
- 1 replies
- 0 kudos
Invalid string or buffer length (0)
I've created a simple query reading all columns from a table. I've published the results on a dashboard, however I receive the following error. I cannot seem to find any info online on how to resolve this issueAny ideas?
- 2997 Views
- 1 replies
- 0 kudos
- 0 kudos
Resolved. I forgot to save the query.
- 0 kudos
- 3342 Views
- 2 replies
- 1 kudos
Resolved! Why are R scripts inside of Databricks notebooks creating writeLines errors?
I recently posted this in Stack Overflow. I'm using R in Databricks. R Studio runs fine and executes from the Databricks cluster. I would like to transition from R Studio to notebooks. When I start the cluster, R seems to run fine from notebooks. ...
- 3342 Views
- 2 replies
- 1 kudos
- 1 kudos
@Paul Evangelista​ - Thank you for letting us know. You did great!Would you be happy to mark your answer as best so that others can find your solution more easily?
- 1 kudos
- 10741 Views
- 2 replies
- 2 kudos
Why database/table names are in lower case ?
Hello,When I run this code :CREATE DATABASE BackOfficeI see the database like this :backofficeWhy everything is in lower case ?Is it possible to configure Databricks in order to keep the real name ?Thanks.
- 10741 Views
- 2 replies
- 2 kudos
- 2 kudos
It is managed by hive metastore as you can put it in different databases is saver this way as some database are Case Sensitive and some not (you can easily test it with standard WHERE syntax).Probably you could change it with some hive settings but i...
- 2 kudos
- 7771 Views
- 1 replies
- 3 kudos
How to download zip file from SFTP location and put that file into Azure Data Lake and unzip there ?
I have zip file on SFTP location. I want to copy that file from SFTP location and put it into Azure Data lake and want to unzip there using spark notebook. Please help me to solve this.
- 7771 Views
- 1 replies
- 3 kudos
- 3 kudos
I would go with @Kaniz Fatma​ approach and download data in Data Factory and after is downloaded on success trigger databricks spark notebook. With spark you can read also compressed data so maybe you will not need to do even separate unzip.
- 3 kudos
- 35589 Views
- 16 replies
- 10 kudos
Resolved! Error in databricks-sql-connector
from databricks import sql hostname = '<name>.databricks.com' http_path = '/sql/1.0/endpoints/<endpoint_id>' access_token = '<personal_token>' connection = sql.connect(server_hostname=hostname, http_path=http_path, access_token=access_token) cu...
- 35589 Views
- 16 replies
- 10 kudos
- 10 kudos
Hi @Borislav Blagoev​ ,Thanks very much for taking the time to collect these logs.The problem here (as indicated by the `IpAclValidation` message) is that IP allow listing (enabled for your workspace) will not allow arbitrary connections from Spark c...
- 10 kudos
- 2018 Views
- 0 replies
- 0 kudos
Snowflake Connector for Databricks
Hi everyone, I am working with Databricks Notebooks and I am facing an issue with snowflake connector, I wanted to use DDL/DML with snowflake connector. Can someone please help me out with this, Thanks in advance !!
- 2018 Views
- 0 replies
- 0 kudos
- 5785 Views
- 3 replies
- 0 kudos
Resolved! Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination
I have a pipeline with + 20 streams running based on autoloader. The pipeline crashed and after the crash I'm unable to start the streams and they fail with one of the following messages:1): The metadata file in the streaming source checkpoint direct...
- 5785 Views
- 3 replies
- 0 kudos
- 0 kudos
@Olli Tiihonen​ - Thanks for letting us know. I'm glad you were able to get to the bottom of things.
- 0 kudos
- 1263 Views
- 0 replies
- 0 kudos
www.nextitvision.com
Online IT Training: ERP/SAP Online Training | JAVA Online Training | C++Online Training | ORACLE Online Training | Online Python Training | Machine Learning Training. If you Need more Details and Information Regarding IT Online Training. Please Visi...
- 1263 Views
- 0 replies
- 0 kudos
- 5820 Views
- 1 replies
- 3 kudos
Resolved! Driver is up but is not responsive, likely due to GC.
Hi,I have a problem with writing an excel file into the mounted file.after 10 mins I see the Driver is up but is not responsive, likely due to GC on the log events.I'm using the following script:df.repartition(1).write .format("com.crealytics.spark....
- 5820 Views
- 1 replies
- 3 kudos
- 3 kudos
It is not solution to that problem but I recommend to handle excel reads and writes with Spark Koalas https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_excel.html just give it a try maybe it will solve your issue
- 3 kudos
- 4165 Views
- 1 replies
- 2 kudos
How can I avoid this 'java.sql.SQLException: Too many connections' error?
I'm having difficulty with a job (parent) that triggers multiple parallel runs of another job (child) in batches (e.g. 10 parallel runs per batch).Occasionally some of the parallel "child" jobs will crash a few minutes in-- either during or immediate...
- 4165 Views
- 1 replies
- 2 kudos
- 2 kudos
It is MariaDB JDBC error so probably database which you are trying to connect can not handle this amount of concurrent connections (alternatively if you are not connecting to MariaDB databse, MariaDB is used also for hive metastore in your case maria...
- 2 kudos
- 1945 Views
- 1 replies
- 1 kudos
Azure Databricks Certification
@Hubert Dudek​ what is the Certification name of Azure Databricks?
- 1945 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Bhagwan Chaubey​ ,There is Spark developer certification from Databricks - https://databricks.com/learn/training/home (and some higher levels as well)In Azure databricks is included in DP-100 and DP-203 certification (together with around 10 diff...
- 1 kudos
- 10891 Views
- 4 replies
- 3 kudos
Resolved! Cost of individual jobs running on a shared Databricks cluster
Hi All,I am working on a requirement where I need to calculate the cost of each spark job individually on a shared Azure/AWS Databricks cluster. There can be multiple jobs running on the cluster parallelly.Cost needs to be calculated after job comple...
- 10891 Views
- 4 replies
- 3 kudos
- 3 kudos
There is a built-in functionality for getting the costs:AWS - https://docs.databricks.com/administration-guide/account-settings-e2/usage.htmlAzure - via built-in Cost Management + BillingThe main problem with that functionality is that the smallest g...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
