- 6644 Views
- 2 replies
- 4 kudos
Azure Databricks Delta live table tab is missing
In my Azure Databricks workspace UI I do not have the tab "Delta live tables". In the documentation it says that there is a tab after clicking on Jobs in the main menu. I just created this Databricks resource in Azure and from my understanding the DL...
- 6644 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Chr Jon​ How are you doing? Thanks for posting your question. Just checking in to see if one of the answers helped, would you let us know?
- 4 kudos
- 3590 Views
- 2 replies
- 2 kudos
Resolved! Using Delta Tables without Time Travel features?
Hi Everyone / Experts,is it possible to use Delta Tables without the Time Travel features? We are primarily interested in using the DML Features (delete, update, merge into, etc)Thanks,Mark
- 3590 Views
- 2 replies
- 2 kudos
- 1345 Views
- 0 replies
- 0 kudos
Reading multiple file and writing final results in Databricks getting below error
Job aborted due to stage failure: Task 12 in stage 1446.0 failed 4 times, most recent failure: Lost task 12.3 in stage 1446.0 (TID 2922) (10.24.175.143 executor 41): ExecutorLostFailure (executor 41 exited caused by one of the running tasks) Reason: ...
- 1345 Views
- 0 replies
- 0 kudos
- 16904 Views
- 2 replies
- 1 kudos
Resolved! How can I access python variable in Spark SQL?
I have python variable created under %python in my jupyter notebook file in Azure Databricks. How can I access the same variable to make comparisons under %sql. Below is the example:%python RunID_Goal = sqlContext.sql("SELECT CONCAT(SUBSTRING(RunID,...
- 16904 Views
- 2 replies
- 1 kudos
- 1 kudos
You can use {} in spark.sql() of pyspark/scala instead of making a sql cell using %sql.This will result in a dataframe. If you want you can create a view on top of this using createOrReplaceTempView()Below is an example to use a variable:-# A variab...
- 1 kudos
- 11426 Views
- 11 replies
- 5 kudos
Resolved! Data bricks cluster creation is failing while running the Cron job scheduling script through init script method from azure data bricks.
Dear connections,I'm unable to run a shell script which contains scheduling a Cron job through init script method on Azure Data bricks cluster nodes.Error from Azure Data bricks workspace:"databricks_error_message": "Cluster scoped init script dbfs:/...
- 11426 Views
- 11 replies
- 5 kudos
- 5 kudos
Hello @Sugumar Srinivasan​ Could you please enable cluster log delivery and inspect the INIT script logs in the below path dbfs:/cluster-logs/<clusterId>/init_scripts path.https://docs.databricks.com/clusters/configure.html#cluster-log-delivery-1
- 5 kudos
- 5648 Views
- 4 replies
- 6 kudos
Resolved! read the csv file as shown in description
Project_Details.csvProjectNo|ProjectName|EmployeeNo100|analytics|1100|analytics|2101|machine learning|3101|machine learning|1101|machine learning|4Find each employee in the form of list working on each project?Output:ProjectNo|employeeNo100|[1,2]101|...
- 5648 Views
- 4 replies
- 6 kudos
- 6 kudos
@SANJEEV BANDRU​ You can simply do thisJust change the file path CREATE TEMPORARY VIEW readcsv USING CSV OPTIONS ( path "dbfs:/docs/test.csv", header "true", delimiter "|", mode "FAILFAST");select ProjectNo, collect_list(EmployeeNo) Employeesfrom re...
- 6 kudos
- 5763 Views
- 5 replies
- 13 kudos
Hello everyone, I have a directory with 40 files. File names are divided into prefixes. I need to rename the prefix k3241 according to the name in the...
Hello everyone, I have a directory with 40 files.File names are divided into prefixes. I need to rename the prefix k3241 according to the name in the last prefix.I even managed to insert the csv extension at the end of the file. but renaming files ba...
- 5763 Views
- 5 replies
- 13 kudos
- 13 kudos
Hi @welder martins​ How are you doing?Thank you for posting that question. We are glad you could resolve the issue. Would you want to mark an answer as the best solution?Cheers
- 13 kudos
- 4251 Views
- 5 replies
- 3 kudos
Is it required to run OPTIMIZE after doing GDPR DELETEs?
Greetings,I have been reading the excellent article from https://docs.databricks.com/security/privacy/gdpr-delta.html?_ga=2.130942095.1400636634.1649068106-1416403472.1644480995&_gac=1.24792648.1647880283.CjwKCAjwxOCRBhA8EiwA0X8hi4Jsx2PulVs_FGMBdByBk...
- 4251 Views
- 5 replies
- 3 kudos
- 3 kudos
@Hubert Dudek​ thanks for the hint, exactly as written in the article VACUUM is required after the GDPR delete operation, however do we need to OPTIMIZE ZSORT again the table or is the ordering maintained?
- 3 kudos
- 4841 Views
- 4 replies
- 2 kudos
Resolved! Random SMTP authentication failures to Office 365 (Exchange)
Hey all-I have a python script running in databricks notebook which uses smtplib to connect and send email via our Exchange online server. At random times, it will start getting authentication failures and I can't figure out why. I've confirmed that ...
- 4841 Views
- 4 replies
- 2 kudos
- 2 kudos
If message is "'bad username or password.'" my guess is that it is on Exchange side.
- 2 kudos
- 8646 Views
- 5 replies
- 7 kudos
Resolved! How to query deltatables stored in s3 through databricks SQL Endpoint?
the delta tables after ETL are stored in s3 in csv or parquet format, so now question is how to allow databricks sql endpoint to run query over s3 saved files
- 8646 Views
- 5 replies
- 7 kudos
- 7 kudos
Hey @Athlestan Jain​ How are you doing?Thanks for posting your question. Do you think you were able to resolve the issue?We'd love to hear from you.
- 7 kudos
- 5178 Views
- 2 replies
- 1 kudos
Resolved! Checkpoint is getting created even the though the microbatch append has failed
Use caseRead data from source table using structured spark streaming(Round the clock).Apply transformation logic etc etc and finally merge the dataframe in the target table.If there is any failure during transformation or merge ,databricks job should...
- 5178 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Om Singh​ Hope you are doing well. Just wanted to check in and see if you were able to find a solution to your question?Cheers
- 1 kudos
- 5507 Views
- 2 replies
- 4 kudos
Resolved! Connecting Delta Tables from any Tools
Hi Team,To access SQL Tables we use tools like TOAD , SQL SERVER MANAGEMENT STUDIO (SSMS).Is there any tool to connect and access Databricks Delta tables.Please let us know.Thank you
- 5507 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Rajesh Vinukonda​ Hope you are doing well. Thanks for sending in your question. Were you able to find a solution to your query?
- 4 kudos
- 3522 Views
- 3 replies
- 2 kudos
Resolved! Py Spark Pandas Code diff
Hi Can you help me why Pandas code not working..but Pyspark is working..import pandas as pdpdf = pd.read_csv('/FileStore/tables/new.csv',sep=',')Error : No such file exists...below is worked..df = spark.read.csv("/FileStore/tables/new.csv", sep=",", ...
- 3522 Views
- 3 replies
- 2 kudos
- 2 kudos
It might has to do with the path as @Hubert Dudek​ already mentioned: df = spark.read.csv("dbfs:/FileStore/tables/new.csv", sep=",", header='True')
- 2 kudos
- 40721 Views
- 6 replies
- 7 kudos
Resolved! Performance for pyspark dataframe is very slow after using a @pandas_udf
Hello,I am currently working on a time series forecasting with FBProphet. Since I have data with many time series groups (~3000) I use a @pandas_udf to parallelize the training. @pandas_udf(schema, PandasUDFType.GROUPED_MAP) def forecast_netprofit(pr...
- 40721 Views
- 6 replies
- 7 kudos
- 7 kudos
Thank you for the answers. Unfortunately this did not solve the performance issue.What I did now is I saved the results into a table:results.write.mode("overwrite").saveAsTable("db.results") This is probably not the best solution but after I do that ...
- 7 kudos
- 2730 Views
- 1 replies
- 2 kudos
Resolved! java.lang.NoSuchMethodError in databricks
I have created a package. Now I am calling a method from this package in my notebook but it is throwing me java.lang.NoSuchMethodError in databricks. The method exists in the package. Can you please guide me regarding the same.Thanks!
- 2730 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi! I am sharing the error stack with you. I can't share the code with you due to confidentiality of the code. Can you please guide me ?java.lang.NoSuchMethodError: com.iig.utils.common.IIGCommonConstants$.flowProperties()Ljava/lang/String; at com.ii...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
