- 11109 Views
- 5 replies
- 2 kudos
Resolved! Formatting on Databricks Alerts
Hi Guys. I have looked at the formatting options and I'm still struggling to work out how to best format the email body of a databricks alert. I want to be able to selectively choose columns from the query and dispaly them in a table. Or even if i ca...
- 11109 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Nick Hughes​ , unfortunately, this is not available for now. We have a feature request for the same. DB-I-4105 - SQL Alerts: Formatting message body when creating Custom TemplateThis feature has been considered by our product team and it will be...
- 2 kudos
- 3822 Views
- 2 replies
- 1 kudos
Resolved! Why EC2 or EBS volumes are not getting tagged for the instances pool
Why EC2 or EBS volumes are not getting tagged for the instances pool when the tags are present for the Clusters
- 3822 Views
- 2 replies
- 1 kudos
- 1 kudos
"If a cluster is created from a pool, its EC2 instances inherit only the custom and default pool tags, not the cluster tags. Therefore if you want to create clusters from a pool, make sure to assign all of the custom cluster tags you need to the pool...
- 1 kudos
- 7139 Views
- 5 replies
- 4 kudos
Resolved! Databricks fails writing after writing ~30 files
Good day, Copy of https://stackoverflow.com/questions/69974301/looping-through-files-in-databricks-failsI got 100 files of csv data on adls-gen1 store. I want to do some processing to them and save results to same drive, different directory. def look...
- 7139 Views
- 5 replies
- 4 kudos
- 4 kudos
was actually anything created by script in directory <my_output_dir>?The best would be to permanently mount ADSL storage and use azure app for that.In Azure please go to App registrations - register app with name for example "databricks_mount" . Ad...
- 4 kudos
- 4324 Views
- 1 replies
- 1 kudos
Resolved! Best practices for working with Redshift
I have a customer with the following question - I'm posting on their behalf to introduce them to the community. For doing modeling in a python environment what is our best practice for getting the data from redshift? A "load" option seems to leave me...
- 4324 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Nicole Wong​ ,Have you check the docs from here? As far as I know, this might be the only way to read/write data to/from redshift.
- 1 kudos
- 16781 Views
- 4 replies
- 4 kudos
Resolved! How does Spark do lazy evaluation?
For context, I am running Spark on databricks platform and using Delta Tables (s3). Let's assume we a table called table_one. I create a view called view_one using the table and then call view_one. Next, I create another view, called view_two based o...
- 16781 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @John Constantine​ ,The following notebook url will help you to undertand better the difference between lazy transformations and action in Spark. You will be able to compare the physical query plans and undertand better what is going on when you e...
- 4 kudos
- 4082 Views
- 2 replies
- 4 kudos
Resolved! Import a notebook in a Release Pipeline with a Python script
Hi, I would like to import a python file to Databricks with a Azure DevOps Release Pipeline.Within the pipeline I execute a python script which contains this code :import sys import os import base64 import requests dbw_url = sys.argv[1] # https://a...
- 4082 Views
- 2 replies
- 4 kudos
- 4 kudos
Recently I wrote about alternative way to export/import notebooks in pthon https://community.databricks.com/s/question/0D53f00001TgT52CAF/import-notebook-with-python-script-using-api This way you will get more readable error/message (often it is rela...
- 4 kudos
- 4080 Views
- 2 replies
- 13 kudos
Resolved! something like AWS Macie to perform scans on Azure Data Lake
Does anyone know alternative for AWS Macie in Azure?AWS Macie scan S3 buckets for files with sensitive data (personal address, credit card etc...).I would like to use the same style ready scanner for Azure Data Lake.
- 4080 Views
- 2 replies
- 13 kudos
- 26756 Views
- 11 replies
- 2 kudos
- 26756 Views
- 11 replies
- 2 kudos
- 2 kudos
Hi @ahana ahana​ ,Did any of the replies helped you solve this issue? would you be happy to mark their answer as best so that others can quickly find the solution?Thank you
- 2 kudos
- 3149 Views
- 2 replies
- 2 kudos
- 3149 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Christopher Shehu​ , if @Piper Wilson​ 's response helped you to solve your question? would you be happy to mark her answer as best so that others can quickly find the solution in the future.
- 2 kudos
- 11705 Views
- 4 replies
- 2 kudos
Resolved! Read JSON with backslash.
Hello guys.I'm trying to read JSON file which contains backslash and failed to read it via pyspark.Tried a lot of options but didn't solve this yet, I thought to read all the JSON as text and replace all "\" with "/" but pyspark fail to read it as te...
- 11705 Views
- 4 replies
- 2 kudos
- 2 kudos
@orian hindi​ - Would you be happy to post the solution you came up with and then mark it as best? That will help other members.
- 2 kudos
- 11053 Views
- 8 replies
- 0 kudos
Hi All, I am trying to read a csv file from datalake and loading data into sql table using Copyinto. am facing an issue   Here i created one table wit...
Hi All,I am trying to read a csv file from datalake and loading data into sql table using Copyinto.am facing an issue Here i created one table with 6 columns same as data in csv file.but unable to load the data.can anyone helpme on this
- 11053 Views
- 8 replies
- 0 kudos
- 0 kudos
Thanks Werners for your Reply,How to pass schema(ColumnName && Types) to CSV file ??
- 0 kudos
- 9936 Views
- 4 replies
- 3 kudos
Resolved! Pyspark logging - custom to Azure blob mount directory
I'm using the logging module to log the events from the job, but it seems the log is creating the file with only 1 lines. The consecutive log events are not being recorded. Is there any reference for custom logging in Databricks.
- 9936 Views
- 4 replies
- 3 kudos
- 3 kudos
@karthick J​ - If Jose's answer helped solve the issue, would you be happy to mark their answer as best so that others can find the solution more easily?
- 3 kudos
- 1499 Views
- 0 replies
- 0 kudos
SQL server error
What is SQL Server 80004005 error? How to fix?
- 1499 Views
- 0 replies
- 0 kudos
- 6024 Views
- 1 replies
- 2 kudos
Resolved! Notebook failing in job-cluster but runs fine in all-purpose-cluster with the same configuration
I have a notebook with many join and few persist operations (which runs fine on all-purpose-cluster (with worker nodes - i3.xlarge and autoscale enabled), but the same notebook failing in job-cluster with the same cluster definition (to be frank the ...
- 6024 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Ajay Nanjundappa​ ,Check "Event log" tab. Search for any spot terminations events. It seems like all your nodes are spot instances. The error "FetchFailedException" is associated with spot termination nodes.
- 2 kudos
- 5242 Views
- 2 replies
- 3 kudos
Resolved! yarn.nodemanager.resource.memory-mb parameter update
I am currently working on determining proper cluster size for my Spark application and I have a question regarding Hadoop configuration parameter yarn.nodemanager.resource.memory-mb. From what I see, this parameter is responsible for setting the phys...
- 5242 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Andriy Shevchenko​ ,Databricks does not use Yarn. I recommend you to try to use Databricks community edition link to get familiar and explore. You can check Ganglia UI to see how is the cluster utilization, memory, cpu, IO, etc
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
