- 4138 Views
- 4 replies
- 10 kudos
Resolved! Date field getting changed when reading from excel file to dataframe
The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/2022.In dataframe it is ...
- 4138 Views
- 4 replies
- 10 kudos
- 10 kudos
Hi Team, @Merca Ovnerud I am also facing same issue , below is the code snippet which I am using df=spark.read.format("com.crealytics.spark.excel").option("header","true").load("/mnt/dataplatform/Tenant_PK/Results.xlsx")I have a couple of date colum...
- 10 kudos
- 2155 Views
- 2 replies
- 0 kudos
Cluster Modes
Given that there are three different kinda of cluster modes, when is it appropriate to use each one?
- 2155 Views
- 2 replies
- 0 kudos
- 0 kudos
Standard clustersA Standard cluster is recommended for a single user. Standard clusters can run workloads developed in any language: Python, SQL, R, and Scala.High Concurrency clustersA High Concurrency cluster is a managed cloud resource. The key be...
- 0 kudos
- 4639 Views
- 1 replies
- 1 kudos
I am new to Databricks and SQL. My CASE statement is not working and I cannot figure out why. Below is my code and the error message I'm receiving. Grateful for any and all suggestions. I'm trying to put yrs_to_mat into buckets.
SELECT *, yrs_to_mat, CASE WHEN < 3 THEN "under3" WHEN => 3 AND < 5 THEN "3to5" WHEN => 5 AND < 10 THEN "5to10" WHEN => 10 AND < 15 THEN "10to15" WHEN => 15 THEN "over15" ELSE null END AS maturity_bucket FROM mat...
- 4639 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Anne-Marie Wood ,I think it's more SQL general issue:you are not comparing any value to `< 3`it should be something like :WHEN X < 3 THEN "under3" SELECT *, yrs_to_mat, CASE WHEN X < 3 THEN "under3" WHEN X => 3 AND <...
- 1 kudos
- 3431 Views
- 5 replies
- 4 kudos
Resolved! Mount Azure Blob Storage with Cluster access control
Hello.I want to mount and share for the one group the container from Azure Blob Storage (It could be simple blob storage or Azure Data Lake Storage gen 2). But I am not able to do it because I am using Cluster with Table Access Control.This is my cod...
- 3431 Views
- 5 replies
- 4 kudos
- 4 kudos
I have a good solution to the problem:I am using Python library.There are some documentation.Topic to be closed.Best regards,Łukasz
- 4 kudos
- 405 Views
- 0 replies
- 0 kudos
Rs 485 IoT Gateway
RS-485 IoT Gateway is used for transmitting data from one device to another usually far away by using a wired LAN or a Wi-Fi. HashStudioz Technologies Inc. provides Smart IoT Gateway Solutions for Businesses like Pharma industries Etc. Our IoT Gatewa...
- 405 Views
- 0 replies
- 0 kudos
- 1518 Views
- 3 replies
- 3 kudos
How do you access a streaming live table's snapshots?
I have read that delta live tables will keep a history of 7 days. However after creating a streaming live table and using the dlt.apply_changes function. With this codedef run_pipeline(table_name,keys,sequence_by): lower_table_name = table_name.l...
- 1518 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Logan Nicol Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 3 kudos
- 5544 Views
- 5 replies
- 0 kudos
Resolved! Can we expose REST API using Databricks python notebook?
We are trying to expose a REST API endpoint using python notebook. Does databricks allow this?
- 5544 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Muniyappan Mani Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 0 kudos
- 1816 Views
- 3 replies
- 2 kudos
I attended "The Best Data Warehouse Is a Lakehouse" on September 20 but have not recieved my voucher although I recieved a mail which mentio...
I attended "The Best Data Warehouse Is a Lakehouse" on September 20 but have not recieved my voucher although I recieved a mail which mentioned below details.@Kaniz Fatma Please help me on same.Mail Id: bose.innovator8@gmail.comPlease complete the ...
- 1816 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Arindam Bose That's great to hear! Please mark the answer as best.Thanks and Regards
- 2 kudos
- 3517 Views
- 3 replies
- 6 kudos
Resolved! Photon Acceleration not getting enabled for ML runtime in Azure
I tried to enable the photon acceleration in ML runtime 9.1 LTS ML (Scala 2.12,Spark 3.1.2) but getting error "selected runtime version does not support photon".I tried for other versions of ML runtime with single and multinode , access mode being s...
- 3517 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Sajid Thavalengal Rahiman Does @Kaniz Fatma response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 6 kudos
- 2793 Views
- 6 replies
- 1 kudos
Delta Live Tables @expect compare tables count between two stages
Hello,I'm wondering if there is an option to make an expectation on DLT that will compare the number of records between two stages and e.g. fail if there is a difference between those counts?I mean something like this:@dlt.table()def bronze(): Some...
- 2793 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Jacek Dembowiak Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 1 kudos
- 1374 Views
- 3 replies
- 1 kudos
I have cleared Databricks Foundational Accreditation today and received the completion certificate. But I didnt receive the badge for the same. By whe...
I have cleared Databricks Foundational Accreditation today and received the completion certificate. But I didnt receive the badge for the same. By when I will receive the badge ?
- 1374 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Sakshi Puchnanda Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 1244 Views
- 3 replies
- 1 kudos
Incorrect results with df.query()
I have tried pulling a single row from a .csv using df.query()However, the data being returned doesn't coincide with the data I'm expecting - it is pulling a different row. Here is my code:df = spark.read.option("header",True).csv(data_fldr + "config...
- 1244 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Peter Ott Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 1711 Views
- 2 replies
- 1 kudos
Multiple DLT Pipelines Sharing a Single Cluster
When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.I understand DLT pipelines utilising their own special clusters...
- 1711 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Bo Zhu Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 1544 Views
- 2 replies
- 0 kudos
Databricks-jdbc and vulnerabilities CVE-2022-42004, CVE-2022-42003
The latest version of Databricks-jdbc available through Maven (2.6.29) now has these two vulnerabilities:https://nvd.nist.gov/vuln/detail/CVE-2022-42004https://nvd.nist.gov/vuln/detail/CVE-2022-42003All due to depending on and including in the jar th...
- 1544 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Lars Joreteg Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 0 kudos
- 2051 Views
- 3 replies
- 1 kudos
Pipeline in Community edition
Hi there,I'm learning Databricks using the community edition. I noticed I don't have way to practice the pipeline in community edition. (The icon below compute). Says I need to upgrade. Is there any way to practice pipeline and follow learning lesson...
- 2051 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @AJ DJ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 1 kudos