- 1904 Views
- 4 replies
- 2 kudos
Resolved! Query RestAPI end point in Databricks Standard Workspace
Do we have option to query delta table using Standard Workspace as a endpoint instead of JDBC
- 1904 Views
- 4 replies
- 2 kudos
- 2 kudos
@somanath Sankaran​ - Would you be happy to mark @Hubert Dudek​'s answer as best if it resolved the problem? That helps other members who are searching for answers find the solution more quickly.
- 2 kudos
- 3028 Views
- 5 replies
- 5 kudos
Resolved! Schema Parsing issue when datatype of source field is mapped incorrect
I have complex json file which has massive struct column. We regularly have issues when we try to parse this json file by forming our case class to extract the fields from schema. With this approach the issue we are facing is that if one data type of...
- 3028 Views
- 5 replies
- 5 kudos
- 5 kudos
Hey there, @Matt M​ - If @Hubert Dudek​'s response solved the issue, would you be happy to mark his answer as best? It helps other members find the solution more quickly.
- 5 kudos
- 4148 Views
- 9 replies
- 3 kudos
Resolved! Tring to create incremental pipeline but fails when I try to use outputMode "update"
def upsertToDelta(microBatchOutputDF, batchId): microBatchOutputDF.createOrReplaceTempView("updates") microBatchOutputDF._jdf.sparkSession().sql(""" MERGE INTO old o USING updates u ON u.id = o.id WHEN MATCHED THEN UPDATE SE...
- 4148 Views
- 9 replies
- 3 kudos
- 3 kudos
Delta table/file version is too old. Please try to upgrade it as described here https://docs.microsoft.com/en-us/azure/databricks/delta/versioning​
- 3 kudos
- 5522 Views
- 3 replies
- 4 kudos
Resolved! How to download zip file from SFTP location and put that file into Azure Data Lake and unzip there ?
I have zip file on SFTP location. I want to copy that file from SFTP location and put it into Azure Data lake and want to unzip there using spark notebook. Please help me to solve this.
- 5522 Views
- 3 replies
- 4 kudos
- 1237 Views
- 1 replies
- 5 kudos
Resolved! We have hundreds of ETL process (Informatica) with a lot of logic pulling various data from applications into a relational db (Target DB). Can we use Delta Lake as the Target DB?
Hi DB Support,Can we use DB's Delta Lake as our Target DB? Here's our situation...We have hundreds of ETL jobs pulling from these Sources. (SAP, Siebel/Oracle, Cognos, Postgres) .Our ETL Process has all of the logic and our Target DB is an MPP syst...
- 1237 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi yes you can the best is to create sql endpoint in premium workspace and just write to delta lake as to sql. This is community forum not support. You can contact databricks via https://databricks.com/company/contact or via AWS, Azure if you have su...
- 5 kudos
- 1609 Views
- 1 replies
- 0 kudos
Invalid string or buffer length (0)
I've created a simple query reading all columns from a table. I've published the results on a dashboard, however I receive the following error. I cannot seem to find any info online on how to resolve this issueAny ideas?
- 1609 Views
- 1 replies
- 0 kudos
- 0 kudos
Resolved. I forgot to save the query.
- 0 kudos
- 1665 Views
- 2 replies
- 1 kudos
Resolved! Why are R scripts inside of Databricks notebooks creating writeLines errors?
I recently posted this in Stack Overflow. I'm using R in Databricks. R Studio runs fine and executes from the Databricks cluster. I would like to transition from R Studio to notebooks. When I start the cluster, R seems to run fine from notebooks. ...
- 1665 Views
- 2 replies
- 1 kudos
- 1 kudos
@Paul Evangelista​ - Thank you for letting us know. You did great!Would you be happy to mark your answer as best so that others can find your solution more easily?
- 1 kudos
- 7404 Views
- 3 replies
- 3 kudos
Resolved! Why database/table names are in lower case ?
Hello,When I run this code :CREATE DATABASE BackOfficeI see the database like this :backofficeWhy everything is in lower case ?Is it possible to configure Databricks in order to keep the real name ?Thanks.
- 7404 Views
- 3 replies
- 3 kudos
- 3 kudos
It is managed by hive metastore as you can put it in different databases is saver this way as some database are Case Sensitive and some not (you can easily test it with standard WHERE syntax).Probably you could change it with some hive settings but i...
- 3 kudos
- 18814 Views
- 16 replies
- 10 kudos
Resolved! Error in databricks-sql-connector
from databricks import sql hostname = '<name>.databricks.com' http_path = '/sql/1.0/endpoints/<endpoint_id>' access_token = '<personal_token>' connection = sql.connect(server_hostname=hostname, http_path=http_path, access_token=access_token) cu...
- 18814 Views
- 16 replies
- 10 kudos
- 10 kudos
Hi @Borislav Blagoev​ ,Thanks very much for taking the time to collect these logs.The problem here (as indicated by the `IpAclValidation` message) is that IP allow listing (enabled for your workspace) will not allow arbitrary connections from Spark c...
- 10 kudos
- 2789 Views
- 3 replies
- 0 kudos
Resolved! Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination
I have a pipeline with + 20 streams running based on autoloader. The pipeline crashed and after the crash I'm unable to start the streams and they fail with one of the following messages:1): The metadata file in the streaming source checkpoint direct...
- 2789 Views
- 3 replies
- 0 kudos
- 0 kudos
@Olli Tiihonen​ - Thanks for letting us know. I'm glad you were able to get to the bottom of things.
- 0 kudos
- 632 Views
- 0 replies
- 0 kudos
www.nextitvision.com
Online IT Training: ERP/SAP Online Training | JAVA Online Training | C++Online Training | ORACLE Online Training | Online Python Training | Machine Learning Training. If you Need more Details and Information Regarding IT Online Training. Please Visi...
- 632 Views
- 0 replies
- 0 kudos
- 2090 Views
- 1 replies
- 2 kudos
How can I avoid this 'java.sql.SQLException: Too many connections' error?
I'm having difficulty with a job (parent) that triggers multiple parallel runs of another job (child) in batches (e.g. 10 parallel runs per batch).Occasionally some of the parallel "child" jobs will crash a few minutes in-- either during or immediate...
- 2090 Views
- 1 replies
- 2 kudos
- 2 kudos
It is MariaDB JDBC error so probably database which you are trying to connect can not handle this amount of concurrent connections (alternatively if you are not connecting to MariaDB databse, MariaDB is used also for hive metastore in your case maria...
- 2 kudos
- 925 Views
- 1 replies
- 1 kudos
Azure Databricks Certification
@Hubert Dudek​ what is the Certification name of Azure Databricks?
- 925 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Bhagwan Chaubey​ ,There is Spark developer certification from Databricks - https://databricks.com/learn/training/home (and some higher levels as well)In Azure databricks is included in DP-100 and DP-203 certification (together with around 10 diff...
- 1 kudos
- 6220 Views
- 5 replies
- 3 kudos
Resolved! Cost of individual jobs running on a shared Databricks cluster
Hi All,I am working on a requirement where I need to calculate the cost of each spark job individually on a shared Azure/AWS Databricks cluster. There can be multiple jobs running on the cluster parallelly.Cost needs to be calculated after job comple...
- 6220 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Ashish Kardam​ (Customer)​ , Does @[werners] (Customer)​ 's or @Alex Ott​ 's replies answer your question?
- 3 kudos
- 10327 Views
- 23 replies
- 13 kudos
Resolved! NoSuchMethodError: org.apache.spark.sql.catalyst.json.CreateJacksonParser on Databricks Cloud (but not on Spark Directly)
Hi,I'm working for Couchbase on the Couchbase Spark Connector and noticed something weird which I haven't been able to get to the bottom of so far.For query DataFrames we use the Datasource v2 API and we delegate the JSON parsing to the org.apache.sp...
- 10327 Views
- 23 replies
- 13 kudos
- 13 kudos
Since there hasn't been any progress on this for over a month, I applied a workaround and copied the classes into the connector source code so we don't have to rely on the databricks classloader. It seems to work in my testing and will be released wi...
- 13 kudos