- 754 Views
- 1 replies
- 0 kudos
Efficient Detection of Schema Mismatch in CSV Files During Single Pass Reading
Hello, when I read a CSV file with a schema object, if a column in the original CSV contains a value of a different datatype than specified in the schema, the result is a null cell. Is there an efficient way to identify these cases without having to ...
- 754 Views
- 1 replies
- 0 kudos
- 0 kudos
Maybe you can try to read the data and let AutoLoader move missmatch data e.g. to rescueColumnhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/schema#--what-is-the-rescued-data-columnThen you can decide what you do with rescue...
- 0 kudos
- 7115 Views
- 5 replies
- 2 kudos
[Unity Catalog]-CosmosDB: Data source v2 are not supported
I've worked on azure databricks connected to azure cosmos. It works when my cluster is not enabling Unity Catalog (UC).But when I enable UC, it return error like below:AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s...
- 7115 Views
- 5 replies
- 2 kudos
- 2020 Views
- 2 replies
- 1 kudos
dbutils.fs.ls versus pathlib.Path
Hello community members,The dbutils.fs.ls('/') exposes the distributed file system(DBFS) on the databricks cluster. Similary, the python library pathlib can also expose 4 files in the cluster like below:from pathlib import Pathmypath = Path('/')for i...
- 2020 Views
- 2 replies
- 1 kudos
- 1 kudos
I think it will be usefull if you look at this documentation to understand difrent files and how you can interact with them:https://learn.microsoft.com/en-us/azure/databricks/files/there is not much to say then that dbutils is "databricks code" that ...
- 1 kudos
- 4398 Views
- 5 replies
- 2 kudos
DLT Compute Resources - What Compute Is It???
Hi there, I'm wondering if someone can help me understand what compute resources DLT uses? It's not clear to me at all if it uses the last compute cluster I had been working on, or something else entirely.Can someone please help clarify this?
- 4398 Views
- 5 replies
- 2 kudos
- 2 kudos
Well, one thing they emphasize in the 'Adavanced Data Engineer' Training is that job-clusters will terminate within 5 minutes after a job is completed. So this could be in support of your theory to lower costs. I think job-cluster are actually design...
- 2 kudos
- 1627 Views
- 1 replies
- 0 kudos
python library in databricks
Hello community members,I am seeking to understand where databricks keeps all the python libraries ? For a start, I tried two lines below:import sys sys.path()This list all the paths but I cant look inside them. How is DBFS different from these paths...
- 1627 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello,all your libraries are installed on Databricks Cluster Driver node on OS Disk.DBFS is like mounted Cloude Storage account.You have veriouse ways of working with libraries but databricks only load some of libraries that comes with Cluster image....
- 0 kudos
- 1780 Views
- 2 replies
- 0 kudos
Seeking Assistance with Dynamic %run Command Path
Hello Databricks Community Team,I trust this message finds you well. I am currently facing an issue while attempting to utilize a dynamic path with the %run command to execute a notebook called from another folder. I have tested the following approac...
- 1780 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, If your config file is in the databricks file system then you should add dbfs:/Ex: f"dbfs:/Users/.../blob_conf/{conf_file}"
- 0 kudos
- 641 Views
- 1 replies
- 0 kudos
Databricks JDBC Driver 2.6.36 includes dependencies in pom.properties with vulnerabilities
Starting from Databricks JDBC Driver 2.6.36 we've got Trivy security report with vulnerabilities from pom.properties.2.6.36 adds org.apache.commons.commons-compress:1.20 and ch.qos.logback.logback-classic:1.2.3.2.6.34 doesn't include such dependencie...
- 641 Views
- 1 replies
- 0 kudos
- 0 kudos
I didn't find where to open an issue (GitHub or Jira). Please, let me know if I need to report it somewhere else.
- 0 kudos
- 4325 Views
- 2 replies
- 2 kudos
UC Volumes - Cannot access the UC Volume path from this location. Path was
Hi, I'm trying out the new Volumes preview.I'm using external locations for everything so far. I have my storage credential, and external locations created and tested. I created a catalog, schema and in that schema a volume. In the new data browser o...
- 4325 Views
- 2 replies
- 2 kudos
- 2 kudos
Hope this helps, but this issue could be caused by the Cluster being in no-isolation shared and not in single-user or shared, both compatible with Unity Catalog
- 2 kudos
- 668 Views
- 0 replies
- 0 kudos
Creating external location is Failing because of cross plane request
While creating Unity Catalog external location from Data Bricks UI or from a notebook using "CREATE EXTERNAL LOCATION location_name .." a connection is being made and rejected from control plane to the S3 data bucket in a PrivateLink enabled environm...
- 668 Views
- 0 replies
- 0 kudos
- 651 Views
- 0 replies
- 0 kudos
Source to Bronze Organization + Partition
Hi there, I hope I have what is effectively a simple question. I'd like to ask for a bit on guidance if I am structuring my source-to-bronze auto loader data properly. Here's what I have currently:/adls_storage/<data_source_name>/<category>/autoloade...
- 651 Views
- 0 replies
- 0 kudos
- 2974 Views
- 2 replies
- 0 kudos
Install python package from private repo [CodeArtifact]
As part of my MLOps stack, I have developed a few packages which are the published to a private AWS CodeArtifact repo. How can I connect the AWS CodeArtifact repo to databricks? I want to be able to add these packages to the requirements.txt of a mod...
- 2974 Views
- 2 replies
- 0 kudos
- 0 kudos
One way to do it is to run this line before installing the dependencies:pip config set site.index-url https://aws:$CODEARTIFACT_AUTH_TOKEN@my_domain-111122223333.d.codeartifact.region.amazonaws.com/pypi/my_repo/simple/But can we add this in MLFlow?
- 0 kudos
- 1313 Views
- 1 replies
- 0 kudos
Databricks JDBC driver multi query in one request.
Can I run multi query in one command using databricks JDBC driver and would databricks execute one query faster then running multi queries in one script?
- 1313 Views
- 1 replies
- 0 kudos
- 0 kudos
Can I run multi query in one command using databricks JDBC driver and would databricks execute one query faster then running multi queries in one script?
- 0 kudos
- 750 Views
- 0 replies
- 0 kudos
DLT pipeline access external location with abfss protocol was failed
Dear Databricks Community Members:The symptom: The DLT pipeline was failed with the error message: Failure to initialize configuration for storage account storageaccount.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account...
- 750 Views
- 0 replies
- 0 kudos
- 1611 Views
- 2 replies
- 0 kudos
Auto Loader Use Case Question - Centralized Dropzone to Bronze?
Good day,I am trying to use Auto Loader (potentially extending into DLT in the future) to easily pull data coming from an external system (currently located in a single location) and organize it and load it respectively. I am struggling quite a bit a...
- 1611 Views
- 2 replies
- 0 kudos
- 0 kudos
Quick follow-up on this @Retired_mod (or to anyone else in the Databricks multi-verse who is able to help clarify this case).I understand that the proposed solution would work for a "one-to-one" case where many files are landing in a specific dbfs pa...
- 0 kudos
- 1877 Views
- 3 replies
- 0 kudos
Fail to write large dataframe
Hi all, we have a issue while trying to write a quite large data frame, close to 35 million records. We try to write it as parquet and also table and none work. But writing a small chink (10k records) is working. Basically we have some text on which ...
- 1877 Views
- 3 replies
- 0 kudos
- 0 kudos
That could work, but you will have to create a UDF.Check this SO topic for more info
- 0 kudos
